Steganographic encoding and decoding

ABSTRACT

This patent document relates generally to steganography and digital watermarking. One claim recites, in a watermark detector, a method of decoding auxiliary information encoded encoding in an image or video. The method includes: receiving data representing the image or video, wherein the data comprises first data corresponding to a first color channel, second data corresponding to a second color channel and third data corresponding to a third color channel; weighting the first data, the second data and the third data according to at least the following two factors: i) a color direction biased toward an anticipated embedding direction; and ii) anticipated image or video distortion introduced to the first data, second data or third data through scanning or signal processing; and determining from weighted first data, weighted second data and weighted third data, changes in an image or video attribute, in which the auxiliary information is conveyed through the changes to sample values representing the image or video. Of course, other claims and combinations are provided too.

RELATED PATENT DATA

This application is a continuation of U.S. patent application Ser. No.11/143,088, filed Jun. 1, 2005 (U.S. Pat. No. 7,668,334), which claimsthe benefit of U.S. Provisional Patent Application Nos. 60/585,490,filed Jul. 2, 2004; 60/620,093, filed Oct. 18, 2004; and 60/642,915,filed Jan. 10, 2005.

The present application is also related to U.S. Pat. Nos. 6,700,995;6,590,996 and 6,307,949.

Each of the U.S. patent documents mentioned in this application ishereby incorporated by reference.

TECHNICAL FIELD

The present invention relates generally to digital watermarking. Inparticular, the present invention relates to modifying imagery to betterreceive digital watermarking.

BACKGROUND AND SUMMARY OF THE INVENTION

Not all images are suitable to receive digital watermarking. Some imagesare too light (e.g., “washed out”) or have a large percentage of pixelsthat include high saturation. Other images may be too dark toeffectively receive digital watermarking. The present invention providesmethods and apparatus to modify an image to better receive digitalwatermarking. The methods and apparatus can be integrated into imageworkflows and only modify those images that may be sub par in terms oftheir ability to receive and hide digital watermarking. One exampleworkflow is image capture for identification documents (e.g., at DMVstation or passport application process). Another example is aphotography studio during its print (or proofs) process. Of course,these are but a few of the many environments in which our inventivetechniques will benefit.

Digital watermarking technology, a form of steganography, encompasses agreat variety of techniques by which plural bits of digital data arehidden in some other object, preferably without leaving human-apparentevidence of alteration.

Digital watermarking may be used to modify media content to embed amachine-readable code into the media content. The media may be modifiedsuch that the embedded code is imperceptible or nearly imperceptible tothe user, yet may be detected through an automated detection process.

There are many processes by which media can be processed to encode adigital watermark. In physical objects, the data may be encoded in theform of surface texturing or printing. Such marking can be detected fromoptical scan data, e.g., from a scanner or web cam. In electronicobjects (e.g., digital audio or imagery—including video), the data maybe encoded as slight variations in sample values. Or, if the object isrepresented in a so-called orthogonal domain (also termed“non-perceptual,” e.g., MPEG, DCT, wavelet, etc.), the data may beencoded as slight variations in quantization values or levels. Theassignee's U.S. Pat. Nos. 6,122,403 and 6,614,914 are illustrative ofcertain watermarking technologies. Still further techniques are known tothose of ordinary skill in the watermarking art.

Digital watermarking systems typically have two primary components: anembedding component that embeds a watermark in the media content, and areading component that detects and reads the embedded watermark. Theembedding component embeds a watermark pattern by altering data samplesof the media content. The reading component analyzes content to detectwhether a watermark pattern is present. In applications where thewatermark encodes information, the reading component extracts thisinformation from the detected watermark.

One problem that arises in many watermarking applications is that ofobject corruption. If the object is reproduced, or distorted, in somemanner such that the content presented for watermark decoding is notidentical to the object as originally watermarked, then the decodingprocess may be unable to recognize and decode the watermark. To dealwith such problems, the watermark can convey a reference signal or“orientation component.” The orientation component is of such acharacter as to permit its detection even in the presence of relativelysevere distortion. Once found, the attributes of the distorted referencesignal can be used to quantify the content's distortion. Watermarkdecoding can then proceed—informed by information about the particulardistortion present.

The Assignee's U.S. Pat. Nos. 6,614,914 and 6,408,082 detail orientationcomponents, and processing methods, that permit such watermark decodingeven in the presence of distortion. In some image watermarkingembodiments, an orientation component comprises a constellation ofquasi-impulse functions in the Fourier magnitude domain, each withpseudorandom phase. To detect and quantify the distortion, the watermarkdecoder converts the watermarked image to the Fourier magnitude domainand then performs a log polar resampling of the Fourier magnitude image.A generalized matched filter correlates a known orientation signal withthe re-sampled watermarked signal to find the rotation and scaleparameters providing the highest correlation. The watermark decoderperforms additional correlation operations between the phase informationof the known orientation signal and the watermarked signal to determinetranslation parameters, which identify the origin of the watermarkmessage signal. Having determined the rotation, scale and translation ofthe watermark signal, the reader then adjusts the image data tocompensate for this distortion, and extracts the watermark messagesignal.

One aspect of the present invention is a method to condition an imageprior to receiving steganographic encoding. The process includesreceiving an image and subjecting the image to an approximation of anexpected workflow process. The subjected image is evaluated to determinewhether the image includes characteristics that are conducive to receivesteganographic encoding in view of the expected workflow process. If thedigital image is not conducive, the image is modified.

Another aspect of the present invention is a method to analyze a digitalimage to determine whether the digital image will be a suitable host toreceive digital watermarking. The method includes processing a digitalimage in accordance with an approximation of an expected workflowprocess, and analyzing the processed digital image to determine whetherthe digital image forms a suitable host to receive digital watermarking.If the digital image is not suitable, the digital image is modified tobetter receive digital watermark in anticipation of the expectedworkflow process.

Additional features, aspects and advantages of the present inventionwill become even more apparent with reference to the following detaileddescription and accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A illustrates an example image workflow.

FIG. 1B illustrates image analysis and modification for the workflow ofFIG. 1A.

FIG. 2 illustrates a response curve for an expected image capture deviceand printer.

FIG. 3A illustrates an image including areas susceptible to washing outwith workflow processing; FIG. 3B illustrates a corresponding luminosityhistogram; and FIG. 3C illustrates the FIG. 3A image after processingaccording to the FIG. 2 curve.

FIG. 4 illustrates tone correction for washed-out images.

FIG. 5A illustrates the FIG. 3A image after applying the tone correctionof FIG. 4; FIG. 5B illustrates a corresponding luminosity histogram; andFIG. 5C illustrates the FIG. 5A image after processing according to theFIG. 2 curve.

FIG. 6A illustrates an image including relatively darker image regions;FIG. 6B illustrates a corresponding luminosity histogram; and FIG. 6Cillustrates the FIG. 6A image after processing according to the FIG. 2curve.

FIG. 7 illustrates tone correction for images including dark or shadowregions.

FIG. 8 illustrates an image including both washed-out regions andrelatively darker image regions.

FIG. 9 illustrates tone correction for an image including bothwashed-out regions and darker image regions.

FIG. 10 is a flow diagram for image analysis.

FIG. 11 illustrates a cumulative histogram of good (or acceptable)images;

FIG. 12 illustrates a cumulative histogram of washed-out images; and

FIG. 13 illustrates a cumulative histogram of images includingrelatively darker regions.

FIGS. 14A and 14B illustrate an image include a relatively high level ofnoise, and FIG. 14C illustrates a corresponding histogram.

FIGS. 15A and 15B illustrate corresponding images—after smoothing of theFIGS. 14A and 14B images—including a relatively lower level of noise,and FIG. 15C illustrates a corresponding histogram.

FIG. 16 illustrates an imaging system characteristic response curve.

FIG. 17 illustrates an image analysis algorithm flow diagram.

FIG. 18 illustrates an image with significant highlight content and acorresponding histogram. The histogram plots image grayscale valuesagainst percentages of pixels in the image.

FIG. 19 illustrates a tone correction calculated for an image (AppendixA, FIG. 3) with significant highlight content.

FIG. 20 illustrates an image with significant highlight content and itshistogram after tone correction of FIG. 19. The histogram shows howpixels values have moved relative to the histogram in FIG. 18.

FIG. 21 illustrates an error rate for a watermark including 96 payloadbits.

FIG. 22 illustrates SNR distributions of two datasets.

FIG. 23 is a diagram showing human contrast sensitivity at variouswatermark embedding spatial frequencies. (FIG. 23 corresponds to FIG. 1in priority application No. 60/620,093, filed Oct. 18, 2004.)

FIG. 24 is a diagram illustrating detection rates for various watermarkdetection weightings for RGB channels. (FIG. 24 corresponds to FIG. 2 inpriority application No. 60/620,093, filed Oct. 18, 2004.)

The above Figures were published in US 2010-0322467 A1, which is herebyincorporated by reference in its entirety.

DETAILED DESCRIPTION

FIG. 1A illustrates a conventional image workflow. A digital image 10 isreceived. For example, the image is captured by a digital camera,converted to a digital image from conventional film, or obtained via anoptical scanning device. The digital image 10 is printed by printer 12onto a substrate 14. For example, if digital image 10 includes aphotograph, it may be printed onto an identification document, such as apassport, driver's license or other photo ID. Printed substrate 14 isexpected to be eventually captured by an image capture device 16 (e.g.,an optical sensor, scanner, digital camera, cell phone camera, etc.).The image capture device 16 produces a digital representation 18 ofsubstrate 14, including a representation of digital image 10 printedthereon.

We can make predictions of resulting image quality based on components(e.g., printer 12 and capture device 16) in a workflow. Moreover, we canmake predictions of a workflow system to even more successfully embedand read digital watermarks.

In an ideal image workflow system all components of a digital watermarkembedder and reader systems are preferably modeled as approximatelylinear, and all watermarked images read in a similar manner (e.g.,assuming equal watermark embedding strength).

In practice, however, a linear model fails due to non-linearityintroduced in capture and print devices. Non-linearity of a capture andprint device can be compensated for by characterizing the devices andcompensating for their response (see, e.g., Assignee's U.S. Pat. No.6,700,995 and U.S. patent application Ser. No. 10/954,632, filed Sep.29, 2004, which are both herein incorporated by reference.).

But in many cases the watermark capture and/or print device are outsidethe control of a watermarking application.

In these cases a captured image is preferably analyzed for its abilityto carry and conceal a watermark, taking into account an anticipatedresponse of an output printer and capture device.

If a predetermined proportion of image pixels are of a level (e.g.,color or grayscale level) that will not be correctly measured orrepresented by the output device/capture device then image correction ormodification is applied. The correction preferably minimizes a visualimpact attributable to watermarking, but still allows an embeddedwatermark to read by ensuring that about 50-85% of the pixels in theimage are in an approximately linear section of a print device. (We mostprefer that at least 70% of the pixels fall within the approximatelylinear section of the print device.).

With reference to FIG. 1B, we insert image analysis 20 and imagemodification 22 modules into the workflow of FIG. 1A. Image analysis 20determines whether the digital image 10, if subjected to an anticipatedprint process (e.g., process 12) and eventual image capture (e.g.,process 16), will provide a suitable environment for digitalwatermarking. If the digital image is acceptable, flow continues to adigital watermark embedder 11, where the digital image is embedded withone or more digital watermarks. Otherwise the digital image is modified22 to provide more favorable image characteristics. A modified digitalimage is provided to the digital watermark embedder 11. The embeddedimage is printed. (While the analysis 20 and modification 22 can bemanually preformed, e.g., via photo editing software like Adobe'sPhotoShop, we prefer an automated the process. For example, the analysis20 and modification 22 can be dedicated software modules andincorporated into an SDK or application program.).

FIG. 2 illustrates a printer/scanner curve, which models anticipatedcharacteristics of (and/or corruption introduced by) printer 12 andscanner 16. We use the printer/scanner curve to estimate how aparticular digital image will respond when subjected to the printer 12and eventual scanning 16. As discussed in Appendix A, a response curvecan be generated by printing a range of test patterns on the printer 12and scanning in the printed test patterns via scanner 16. The scanned intest patterns are analyzed to model the workflow process (e.g., printer12 and scanner 16). Results of this process are represented in theresponse curve FIG. 2 (e.g., in terms of luminance, or defining a rangeof pixel values that fall within a linear region). In otherimplementations we represent the workflow response as a data file orequations, etc.

To illustrate further consider the image shown in FIG. 3A. After passingthrough (or analysis relative to) the printer/scanner response curve(FIG. 2), a resulting image will appear washed-out or faded as shown inFIG. 3C. A luminosity histogram corresponding to FIG. 3A is shown inFIG. 3B. The percentage of pixels in the range from low_lum to high_lum(e.g., defined by the range shown between dashed lines in FIG. 2) isonly about 16% in this case. The large number of pixels in the highlightor high luminance areas creates an image which appears washed-out. Thisimage will have a hard time effectively hosting digital watermarking,once it passes through the expected print/scan process, since a highnumber of its pixels correspond to a non-linear portion of the printercurve (FIG. 3B). To compensate for these anticipated problems, tonecorrection is applied to bring a predetermined number of image pixelsinto the linear region of the printer/scanner (e.g., between the dashedlines shown in FIG. 2). While we prefer to bring at least 70% of thepixels within this linear region, we can accept a range having a lowerboundary of approximately 50% of pixels within the linear region.

We correct the tone of the image shown in FIG. 3A according to a curveshown in FIG. 4. The dashed line in FIG. 4 represents preferred imagetone, while the solid line represents the actual image tone. The FIG. 3Aimage is tone corrected, e.g., pixel values are reduced. This processmoves the solid line more toward the dashed line in FIG. 4. This tonecorrection process brings a predetermined number of pixels (e.g., 70%)into the desired linear range. For example, a predetermined number ofpixels now fall between the dashed lines of FIG. 2. The tone correctionis preferably applied to the red, green and blue channels (or CYMKchannels, etc.). Tone correction can be automated using conventionalsoftware such as Adobe's PhotoShop. Of course, software modules (e.g.,C++) can be written to accomplish the same task.

A resulting image—after tone correction—is shown in FIG. 5A. We pass themodified FIG. 5A image through (or analyze relative to) theprinter/scanner response curve (FIG. 2) to determine whether themodified image (FIG. 5A) will provide a sufficient watermark hostingenvironment when subjected to printer 12 and scanner 16. A resultingimage—after passing through the response curve—is shown in FIG. 5C. TheFIG. 5A image is deemed to have acceptable characteristics. (FIG. 5Aluminosity features are shown in FIG. 5B. A comparison between FIG. 5Band FIG. 3B helps to illustrate that the overall highlight pixels inFIG. 5A have been reduced compared to FIG. 3A). We determine that theFIG. 5C image is better able to host a watermark—in comparison to FIG.3C—and is also more visually pleasing.

Correction can improve images with larger shadow regions also. The imagerepresented in FIG. 6A is passed through (or analyzed relative to) theprinter/scanner response curve (FIG. 2), which results in the FIG. 6Cimage. (FIG. 6A luminosity is shown in FIG. 6B.). We determine that weneed to move pixels from the low luminance (or shadow region) into amore linear region of the printer/scanner. For this case, a shadow tonecorrection is performed as shown in FIG. 7. The solid line representingimage pixels values are changed to move the pixel values toward thedashed line.

The above two situations (washed out and darker regions) are sometimesfound in the same image. For example, consider FIG. 8, which includes acombination of darker and lighter areas. Again the FIG. 2printer/scanner curve is used for evaluation, and again we determinethat correction is needed. This time we apply tone correction shown inFIG. 9. This correction moves pixel value from both highlights andshadows toward the preferred dashed line.

FIG. 10 illustrates a process to help determine whether correction isneeded for a particular image. This process can be automated usingsoftware and/or hardware. An image is analyzed to determine pixelvalues. The process determines whether a predetermined number of pixelvalues fall within a shadow region (e.g., as illustrated, <=15%). Theshadow region is preferably identified by a value of low luminance(e.g., see FIG. 2) relative to an expected workflow process. This is thelower level at which the system exhibits non-linear behavior. If notwithin this limit, shadow tone correction is applied. In a first andpreferred implementation, all image pixels are tone corrected. In asecond implementation only a subset of image pixels are tone corrected.For example, only those pixels having predetermined pixel value (here,at this stage, a low pixel value) are adjusted. Similarly, it isdetermined whether a predetermined number of pixels fall withinpredetermined midtone ranges (e.g., as illustrated, >=70%). These rangesare preferably determined by reference to the response curve of FIG. 2(e.g., the area falling within the two dashed lines.) If not, highlighttone correction is applied. As discussed above, we prefer that all imagepixels are tone corrected.

A predetermined number of pixels can be determined according to a numberof factors, e.g., visibility constraints, watermark detection ratesneeded, aesthetic considerations, etc. If modification is preformed, theprocess can be repeated to ensure desired image characteristics. (Imagemodification is preferably automatically applied prior to watermarkembedding, which results in more consistent watermark detection forotherwise poor images.)

Returning to FIG. 1B, an acceptable image 10—an image including apredetermined number of pixels falling within a reasonable tonal rangerelative to an expected response of a workflow—passes unchanged to theprinter 12.

In order to further illustrate the differences between histograms ofgood (FIG. 11), washed out (FIG. 12) and dark (FIG. 13) tonal rangeimages, cumulative histograms are plotted for a range of these imageswith error bars to show spread of data. A cumulative histogram is anaccumulated sum of a number of pixels in each luminance bin. Tenluminance bins from black to white are on the x axis, with thecumulative percentage of the pixels in the image on the y axis. Forexample, in FIG. 13, between 80% and 100% luminance, <=15% of the pixelsare preferred to be in that region. In fact about 70% of the pixels arein that region (1.0-0.3) so a tone correction (FIG. 7) is performed tochange the number of pixels in the highlight region to be <=15%.

While we have made image changes based on luminance or tonalityconsiderations, we can also apply our techniques to other imagecriteria. For example, if an image is too busy or noisy (or,alternatively, not busy or noisy enough)—relative to the business ornoisiness of a typical image—we can modify the image to better receivedigital watermarking in anticipation for the expected printer andscanner (or in anticipation for expected signal processing, likefiltering, etc.). FIGS. 14A and 14B illustrate a noisy image, which issmoothed (FIGS. 15A and 15B) prior to watermark embedding and printing.(The term “smoothed” implies a filtering process, like a Gaussian orWeiner filter, or wavelet based filter.). Corresponding histograms areshown in FIGS. 14C (noisy image) and 15C (smoothed image) respectively.

Another image analysis identifies compression artifacts that areanticipated to be introduced in a later workflow process. We correct animage to compensate for such anticipated artifacts, prior to watermarkembedding.

Another implementation of our invention allows a user to identify anexpected workflow process. A library of response curves is queried toidentify a corresponding response curve for the identified workflowprocess. A received digital image is analyzed relative to thecorresponding response curve.

CONCLUSION

The foregoing are just exemplary implementations of the presentinvention. It will be recognized that there are a great number ofvariations on these basic themes. The foregoing illustrates but a fewapplications of the detailed technology. There are many others.

The section headings in this application are provided merely for thereader's convenience, and provide no substantive limitations. Of course,the disclosure under one section heading may be readily combined withthe disclosure under another section heading.

To provide a comprehensive disclosure without unduly lengthening thisspecification, each of the above-mentioned patent documents are herebyincorporated by reference. The particular combinations of elements andfeatures in the above-detailed embodiments are exemplary only; theinterchanging and substitution of these teachings with other teachingsin this application and the incorporated-by-reference patent documentsare expressly contemplated.

Additional digital watermarking methods and systems are disclosed inAppendix A, which is hereby incorporated by reference. Appendix Aprovides even further discussion of the methods, systems and apparatusdiscussed in the body of this application.

Many of the above-described methods and related functionality can befacilitated with computer executable software stored on computerreadable media, such as electronic memory circuits, RAM, ROM, EPROM,flash memory, magnetic media, optical media, magnetic-optical media,memory sticks, hard disks, removable media, etc., etc. Such software maybe stored and/or executed on a general-purpose computer, or on a serverfor distributed use. Also, instead of software, a hardwareimplementation, or a software-hardware implementation can be used.

In view of the wide variety of embodiments to which the principles andfeatures discussed above can be applied, it should be apparent that thedetailed embodiments are illustrative only and should not be taken aslimiting the scope of the invention.

APPENDIX A Automatic Pre-Processing after Image Robustness Analysis 1.Introduction

A characteristic response curve of an imaging system is measured, todetermine system linearity. Images to be watermarked are analyzed tocheck if a prescribed target minimum of pixels (e.g., 70% or more of thepixels) are in a linear section of the imaging system response. Whenthis criterion is met, the image data is left unchanged. However when itis not met, a tone correction is calculated and applied to the imagedata so the prescribed target minimum of the pixels are in the linearportion. The automatic pre-processing of these problem images increasestheir robustness and reduces variance in robustness of a set ofwatermarked images.

Within the same class of media some Works are much more amenable todigital watermarking than are others. A concrete example of this occursfor the class of watermarking techniques based upon correlation. For thecase of an embedder of Cox, Miller, and Bloom (see documents mentionedat the end of Appendix A), a different watermark gain is applied to eachindividual Work in a set of images according to the level ofpre-existing correlation between the Work and the watermark pattern thatis to be embedded.

It is more difficult to make the kind of informed embedding modificationmentioned above in watermarking systems that require printing and imagere-acquisition. Typically one seeks to appropriately model a combinedprint/image acquisition channel in terms of how it affects the abilityof a given Work to hold a watermark. The model can be used to improvethe efficacy of the system by adaptively changing the applied watermarkin some fashion as in informed embedding.

In this Appendix A we describe a two act procedure for imagewatermarking in environments where images are printed and ultimatelyreacquired using an image capture device such as a scanner or digitalcamera. The first act includes pre-conditioning the image, if necessary,so that it is more suitable for watermarking. The second act involvesembedding the watermark as normal. One example of preprocessing mediaprior to watermarking was presented by Cox, et. al (see referencesmentioned at the end of Appendix A). As will be appreciated, the Cox, etal. type of preprocessing, and the purpose of its employment in theirapplication, is different than that described herein. In the nextsection we develop the need for preprocessing as it applies to thewatermarking system of interest. The details of the algorithm areprovided in section 3. Section 4 deals with experiments andcorresponding results.

2. Need for Preprocessing

Non-linear amplitude modifications that occur as a result of documentprinting and re-acquisition can pose problems of consistency inwatermarking applications. Such non-linearities can be compensated forby characterizing the devices and making adjustments to counteract theireffects. For example many manufacturers use color profiles to correctthe response of the printer and image capture device (see referencesmentioned at the end of Appendix A). However in many cases a watermarkcapture and/or print device are not adequately corrected and are outsidethe control of the watermarking application. In this case the capturedimage can be analyzed for its ability to hold a watermark, taking intoaccount the response of the capture device and output printer.

In order to characterize the non-linearities in the system a set of graypatches that linearly span the range from 0 to 255 (black to white) areoutput using the target printer and substrate. The patches are thenre-acquired (e.g., optically scanned) and an average luminance of eachpatch is measured. The response of an example system is shown in FIG.16. As is usually the case, saturation is seen in the highlight andshadow regions of the imaging system response curve, with anapproximately linear portion that covers a range between points labeledlow gray and high gray.

It can be seen from FIG. 16 that large modifications to a digital imagein areas less than low gray or greater than high gray are mapped tosmall effective changes in the reacquired image. This implies thatimages with a large concentration of pixels in the low gray and/or highgray areas tend to require a higher watermark gain in order to achievethe same robustness as images with smaller concentrations in theseregions. For the purpose of our discussion we refer to an additivespatial domain spread spectrum watermark, but the same comments apply toother watermark types as well.

From a system perspective, where it may be desirable to achieve acertain robustness target for a large set of documents, global watermarkstrength is set high so that the watermark can be recovered for the caseof those images that contain an unusual number of pixels in thenon-linear regions. This would be done to address individual cases thatsuffer from saturation. An example application is watermarking identitydocuments where a very large number of digital images from disparatesources must be dealt with. A problem with using a global watermark gainfor all images in order to satisfy a robustness requirement is that mostimages will be watermarked more strongly than is necessary.

A better solution is to customize the application of the watermark foreach image in the set. Although it is possible to increase watermarkgain in problem areas, a better approach is to adjust low gray/high grayregions to lie within a linear range of a printer/re-acquisition systemprior to applying digital watermarking.

3. Image Tone Correction Algorithm

In order to deal with images that have significant saturation problems,we have developed methods and systems that applies a tone correction tobring about 70% of the pixels into the linear region of theprinter/re-acquisition response curve. To ensure that about 70% of thepixels in an image fall in the approximately linear region, theprocedure shown in FIG. 17 is used. Images without significantsaturation problems would pass through the procedure unchanged. Forpredominantly light or dark images, the correction is automaticallyapplied prior to watermarking resulting in more consistent watermarkdetection. For color images, the image is converted to grayscale todetermine the amount of correction to be applied; then the samecorrection is applied to all channels.

An example of an image with saturation problems in the highlights isshown in FIG. 18 with its corresponding histogram. In the absence oftone correction, this image would require large (or intense) watermarksignal gain in order for the watermark to be detected after the image ispassed through the system characterized by FIG. 16. FIG. 19 depicts thetone correction that is applied to the image when it is subjected to themethod described in FIG. 17. The resulting corrected image and histogramare shown in FIG. 20

The overall effect of the described pre-processing methods is to reducethe contrast in problem images. If a saturation problem lies mainly withthe printer, the algorithm preferably has a negligible visual impact.If, on the other hand, a capture device is the main limiter, the imagecontrast will appear reduced. A tradeoff here is whether noise due towatermarking for all images is more or less objectionable than reducedcontrast in some images.

4. Experiments and Discussion

We conducted an experiment in which a set of 40 images, typical ofidentity document portraits, were embedded at constant watermarkstrength after applying the tone correction algorithm described in theprevious section. Results are compared against a control set whereas thesame set of images were embedded using the same watermark strength inthe absence of preprocessing. We used an HP DeskJet 970Cxi with highquality setting to print the images on Hammermill paper. The images wereacquired using an HP scanner at 300 dpi. The imaging system used for ourexperiment has a characteristic response curve with fewer saturationproblems than we would expect most practical systems to have.

The figure of merit we used to evaluate each dataset is its empiricaldistribution of signal to noise ratio (SNR). For the watermarking systemconsidered in this paper, the ultimate step in watermark reading ispayload recovery, which inevitably requires error correction decoding.For even more details please see the cited references [4], [5]. SNR is ameasure of strength of the recovered coded payload bits from a givenimage just prior to being submitted to the decoder. Each such SNRmeasurement corresponds to a probability of making errors in thedecoding of an N bit watermark. An instance of this correspondence isshown in FIG. 21 for a payload of 96 bits. The figure shows that if theSNR for a particular case is −1 dB, the probability of not decoding thepayload is roughly 10%.

In general, SNR distributions with higher mean and smaller variance leadto better robustness. Of particular importance is the portion of the SNRdistribution lying below the mean since this section affects robustnessthe most.

The relationship between a dataset's SNR distribution and its overallerror rate (r) is given by the following equation:

$\begin{matrix}{r = {\sum\limits_{i = 1}^{M}{{p\left( {SNR}_{i} \right)}{d\left( {SNR}_{i} \right)}}}} & (1)\end{matrix}$

In equation (1), the variable i indexes all M non-zero values of adatasets' SNR distribution, which is represented by p(SNR_(i)). Thevariable, d(SNR_(i)) is the probability of not decoding the payload atthe SNR level indexed by i. In general, SNR distributions with highermean and smaller variance have better robustness. Of particular impactis the portion of the SNR distribution lying below the mean because itaffects robustness the most.

The empirical distributions of SNR for the control and curve correcteddatasets are shown in FIG. 22. By applying equation 1 to the twodatasets we find that the control dataset's error rate is approximately6⊙10⁻⁷, while that of the curve corrected dataset is <10⁻⁹. Thedifference in the two error rates is due to the fact that the lower tailof the control dataset's distribution is significant in extent whilethat of the curve corrected set is minimal.

5. Conclusions

By measuring the characteristic response curve of an imaging system andanalyzing images to check if about 70% or more of the pixels in theimage are in the linear portion, problem images for watermarking can beidentified. These problem images are pre-processed by applying a tonecorrection to the image data so that at least 70% of the pixels are inthe linear portion of the imaging system. Images that already have 70%or more of the pixels in the linear portion are left unchanged.

The automatic pre-processing of these problem images increases therobustness of the watermark in this case. A set of images were processedin this manner, and the watermark robustness was compared to the casewhen no pre-processing was applied. The automatic pre-processing ofthese problem images increases their robustness and reduces the variancein robustness of a set of watermarked images.

The use of this method enables a required detection rate to be obtainedat significantly lower watermark strength, thus reducing watermarkvisibility. Alternatively, by pre-processing the outlier images asignificantly higher detection rate can be achieved at approximately thesame visibility.

6. Additional References

Each of the following documents is herein incorporated by reference: [1]Cox, I. J., Miller, M. L., and J. A. Bloom, Digital Watermarking, MorganKaufmann Publishers, San Diego, 2002; [2] I. J. Cox, M. L, Miller,“Preprocessing media to facilitate later insertion of a watermark,” DSP2002, Santorini, Greece, Jul. 1-3, 2002; [3] D. McDowell, “ColorManagement: What's New from The ICC,” The Pre-Press Bulletin, pp. 4-8,November/December 2001; [4] A. M. Alattar, “‘Smart Images’ UsingDigimarc's Watermarking Technology,” Proc. IS&T/SPIE's 12thInternational Symposium on Electronic Imaging, vol. 3971, number 25, pp.264-273, January 2000; and [5] B. A. Bradley and H. Brunk, “Comparativeperformance of watermarking schemes using M-ary modulation with binaryschemes employing error correction coding,” Proc. SPIE ElectronicImaging, Security and Watermarking of Multimedia Content III, vol. 4314,San Jose, Calif., January 2001.

APPENDIX B (US PROVISIONAL APPLICATION NO. 60/620,093) WatermarkEmbedding and Detection in Optimized Color Space Direction RELATEDAPPLICATION DATA

This application is related to the following U.S. Patent documents: U.S.Pat. Nos. 6,804,377, 6,763,124, 6,700,995, 6,633,654, 6,614,914.6,590,996, 6,122,403 and 5,862,260, which are each hereby incorporatedby reference.

TECHNICAL FIELD

The invention relates to image processing and specifically relates tocolor image processing.

BACKGROUND AND SUMMARY

In color image processing applications, it is useful to understand howhumans perceive colors. By understanding the human visual system and itssensitivity to certain colors, one can more effectively create andmanipulate images to create a desired visual effect, This assertion isparticularly true in image processing applications that intentionallyalter an image to perform a desired function, like hiding information inan image or compressing an image. In digital watermarking, for example,one objective is to encode auxiliary information into a signal, such asan image or video sequence, so that the auxiliary information issubstantially imperceptible to humans in an output form of the signal.Similarly, image compression applications seek to decrease the amount ofdata required to represent an image without introducing noticeableartifacts.

A useful tool in watermark embedding and reading is perceptual analysis.Perceptual analysis refers generally to techniques for evaluating signalproperties based on the extent to which those properties are (or arelikely to be) perceptible to humans (e.g.. listeners or viewers of themedia content). A watermark embedder can take advantage of a HumanVisual System (HVS) model to determine where to place an image watermarkand how to control the intensity of the watermark so that chances ofaccurately recovering the watermark are enhanced, resistance totampering is increased. and perceptibility of the watermark is reduced.Similarly, audio watermark embedder can take advantage of a HumanAuditory System model to determine how to encode an audio watermark inan audio signal to reduce audibility. Such perceptual analysis can playan integral role in gain control because it helps indicate how watermarkembedding can be adjusted relative to the impact on the perceptibilityof the mark. Perceptual analysis can also play an integral role inlocating the watermark in a host signal. l;or example, one might designthe embedder to hide a watermark in portions of a host signal that aremore likely to mask the mark from human perception.

The invention relates to selective color digital watermark emheding ofimages.

Digital watermarking is a process for modifying physical or electronicmedia to embed a machine-readable code into the media. The media may bemodified such that the embedded code is imperceptible or nearlyimperceptible to the user, yet may be detected through an automateddetection process. Most commonly, digital watermarking is applied tomedia signals such as image signals, audio signals, and video signals.However, it may also be applied to other types of media objects,including documents (e.g.. through line, word or character shifting),software, multi-dimensional graphics models, and surface textures ofobjects.

Digital watermarking systems typically have two primary components: anencoder that embeds the watermark in a host media signal, and a decoderthat detects and reads the embedded watermark from a signal suspected ofcontaining a watermark (a suspect signal). The encoder embeds awatermark by subtly altering the host media signal. The physicalmanifestation of altered host media most commonly takes the form ofaltered signal values, such as slightly changed pixel values, imageluminance, color or color values, picture colors, altered DCT ortransform domain coefficients, etc. The reading component analyzes asuspect signal to detect whether a watermark is present. In applicationswhere the watermark encodes information, the reader extracts thisinformation from the detected watermark.

Several particular watermarking techniques have been developed. Thereader is presumed to be familiar with the literature in this field.Particular techniques for embedding and detecting imperceptiblewatermarks in media signals are detailed, e.g.. in assignee'sabove-cited patent documents.

A watermark encodes auxiliary information in an image by modifyingattributes of image samples in the image. As part of the encodingprocess, the encoder computes a change in an image attribute, such asluminance, to an image sample to encode auxiliary information in theimage. The encoder changes color values of the image sample to effectthe change in luminance with minimized impact on visibility.

One aspect of the invention provides a method of encoding auxiliaryinformation in an image. The method includes computing a change in anattribute of an image sample to encode auxiliary information in theimage; and changing color values of the image sample to effect thechange in the image sample attribute. The changes to color values aredetermined based at least in part on both: i) visibility of the changes,and ii) anticipated watermark detection.

Another aspect of the invention is a method of decoding auxiliaryinformation encoded in an image. The method includes receiving opticalscan data corresponding to the image. The scan data includes first datacorresponding to a first color channel, second data corresponding to asecond color channel and third data corresponding to a third colorchannel. The method further includes weighting the first data, thesecond data and the third data according to at least the following twofactors: i) a color direction anticipated to correspond with theembedding; and ii) anticipated image distortion introduced to the firstdata, second data or third data through optical scanning or signalprocessing. The method then includes determining from the weighted firstdata, second data and third data, changes in an image sample attribute,wherein the auxiliary information is conveyed through the changes.

According to still another aspect, a watermark embedding methodincludes: i) receiving an image including a generally white region; ii)determining a color change to embed a portion of a watermark signal inthe white region in terms of a yellow channel; and iii) offsetting theyellow channel change in the white region with color changes in the cyanand magenta channels of the white region.

Further features of the invention will become even more apparent withreference to the following detailed description and accompanyingdrawings.

DETAILED DESCRIPTION

Two constraints may influence digital watermark embedding: 1) HVS; and2) watermark detection. Digital watermarks are preferably embedded toreduce or minimize visibility of the watermark. A HVS model can be usedto guide embedding to reduce visibility. A watermark can also beembedded to ease watermark detection. If an image sensor used inwatermark detection is known to loose information (e.g.. throughcompression or noise) in a particular color channel, a watermark can beembedded in certain color channels to combat the information loss.

Relative sensitivity of the human visual system (HVS) to luminance,red/green or blue/yellow signals often depends on a spatial frequency ofthe signals. Consider a sine wave. If a sine wave is added to a flatgray RGB image, the relative weightings of R:G:B to minimize visibilityto the HVS is dependent on the spatial frequency of the sine wave. Torexample, adding signal changes equally (1:1:1) into the flat grey RGBimage, to accommodate the sine wave, will create a luminance change thatmaximizes the visibility of the signal. An unequal distribution of thesine wave between the RGB color channels will result in a more visiblyappealing image.

If HVS factors are the only considerations for watermark embedding, thenone may be tempted to embed in a yellow channel, since human sensitivityto yellow is very low. Bui real world constraints force consideration ofother factors -- like image sensors. (Remember that a typical watermarkdetection process analyzes image data collected from an optical sensor.)

In an ideal world, where optical sensors (sometimes hereafter referredto as “image sensors”) are “ideal.” a watermark reader is preferablyconfigured to look in the same color direction as the hidden watermarksignal is embedded. This maximizes detectability. For example, if aprinted image includes a watermark embedded only in a yellow channel,then an watermark detector preferably reads the watermark using imagedata only from a blue channel, since increasing the amount of yellow ink(via the watermark in the yellow channel) results in more light beingabsorbed (detectable in the blue channel).

As the quality of an image sensor moves away from ideal, other factorsfor watermark embedding are preferably considered to ensure a balancebetween visibility and detectability. For example, it may not befeasible to sample data only from the blue channel. Many of today'sdigital cameras and image sensors have a lower sampling frequency inblue and red. than in green (due. e.g.. to a Bayer pattern imagingapproach). The blue channel is also a favor target for compressionalgorithms—often resulting in information loss in the blue channel.Noise and channel distortion may also affect detectability.

Thus, while embedding a watermark only in a yellow channel might providefavorable results in terms of visibility, detection of such yellowchannel watermarks may be hampered by real world constraints.

Accordingly, we preferably distribute our watermark signal across colorchannels of an image, while primarily providing our watermark signal inthe blue-yellow direction. This embedding technique minimizes visibilityof the signal, especially at a higher spatial frequency.

Embedder Optimization

A similar argument can be used with a digital watermark which has acertain dominant spatial frequency (represented in terms of “wpi”). Adominant spatial frequency for a 25 wpi, 75 wpi and 150 wpi watermarkare shown in FIG. 23 above for a printed image with a viewing distanceof 9-12 inches. The difference in visibility between embedding in theblue/yellow direction or the luminance direction is strongly dependenton the dominant spatial frequency or wpi of the watermark. For examplewhen embedding at 25 wpi the difference in human contrast sensitivity isabout a factor of 2, whereas at 75 wpi it is about a factor of 15.

An additional constraint is that a watermark is preferably delectable bya grayscale scanner. Thus, the optimum RGB weightings for inserting anenergy watermark (e.g.. energy in the grayscale domain), whileminimizing visibility to the human eye. depend on this relativesensitivity. In the 75 wpi case this can be approximately achieved byhaving a ratio of 1:1:15 for the R:G:B watermark energy (see discussionof 75 wpi in the above paragraph for the 15 weightings).

The arguments above for RGB images also apply to printed CMYK images,where cyan corresponds to red. magenta to green and yellow to blue andblack is left unchanged.

One method calculates a watermark tweak (or changes to an attribute of ahost image, e.g.. at a particular spatial area or frequencyrepresentation), and determines a ratio for CMYK printing ink torepresent the changes. We note that attribute changes can be representedin terms of color changes. We prefer that the color changes preserve theoriginal attribute changes. For example, if the attribute is luminance,the color changes to different channels of the image preferable preservethe corresponding change in luminance.C′=C+K−C*K/255,  (1)where C′ is an ink value with black included, and C is ink valueswithout black included. M′ and Y′ values are readily obtainable from theabove example.C−CtweakA=(C−Ctweak)+K−(C−Ctweak)*K/255.  (2)where CtweakA is a tweak (or watermark value adjustments) with blackincluded to produce a luminance tweak deltaL. MtweakA and YtweakA followaccordingly.Subtracting (1) from (2) yields:CtweakA=Ctweak−Ctweak*K/255.  (3)Required tweak deltaL is a weighted sum of the C, M and Y tweak withblack includeddeltaL=0.2*CtweakA+0.5*MtweakA+0.2* YtweakAThen to obtain ratio of 16:1:1 for Y:M:C =>YtweakA =16 CtweakA,MtweaKA=CtweakAdeltaL=0.3*CtweakA+0.5*CtweakA+0.2*16*CtweakACtweakA=deltaL/4.  (4)Substituting (3) in (4) yields:deltaL/4=Ctweak*(1−K/255)Ctweak=(deltaL/4)/(1-K/255)Related calculations for magenta and yellow yield:Mtweak=(deltaL/4)/(1-K/255)Ytweak=deltaL/(1-K/255).

In our preferred implementation, we represent an attribute change withfour times the yellow relative to the change in the cyan and magentachannels. We have also found acceptable ranges of 16-2 times the yellowchannel relative to the change in the cyan and magenta channels. We ofcourse recognize that this range may vary given different detectionchannel characteristics.

For an ideal image sensor the ratio of signal in the yellow channel ispreferably maximized since this minimizes the visibility of a watermarkof given energy (measured in terms of variation of luminosity). Aproblem that occurs in practice when embedding in yellow is when thesignal is railed by reaching 0% or 100% ink value. Visually this effectis severest close to while, where the positive tweaks increasing yellowink outnumber the negative tweaks which are railed by reaching 0% ink.The result is a highlight with a yellow cast or hue (compared to a graycast with a luminance tweak). One approach avoids this problem bydetermining an amount of yellow that is railed (e.g., an amount ofyellow ink that could not be subtracted) and add this amount as cyan andmagenta ink to the tweaked image region to maintain approximately thecorrect color balance. The addition of cyan and magenta can be added,e.g.. as noise or background coloration. To a first approximation theadded cyan and magenta coloration does not interfere with the detectionprocess which selects mostly the blue channel of the sampled image data.

A similar railing effect occurs close to 100% where the negative tweaksoutnumber the positive tweaks that arc railed by reaching 100% ink. Thisblue cast can also be corrected in a similar manner, hut is often lessobjectionable visually and may not require correction.

An adaptive embedding method could change from mainly yellow embeddingto a more neutral color balance as yellow tends towards to zero. Thedetector would be adaptive in a similar manner, but implies less thanideal detection in the highlights since definition of 0% yellow/bluewould probably be different in the detector.

In one implementation, optimum color channel weightings (R:G:B) for acorresponding image sensor system is determined empirically orcalculated and then checked experimentally. In order to determineweightings empirically a reference, captured image is passed through thedetector using several different sets of weighting coefficients goingfrom a matched set with the ratio 1:1:15 to others which reduce the bluecomponent (to reduce noise due to JPEG compression) or increase thegreen component to take advantage of the better camera resolution ingreen (due to Bayer pattern). The results are used to guide rations forboth embedding and detecting.

Detector Optimization

Watermark detection using an ideal image sensor, which has low noise andequal sensitivity in red, green and blue, is preferably matched to theembedding color direction. This ideal assumption, and resulting matchingassumption, unfortunately begins to fail when “real” sensors arcconsidered. Tor example, while this ideal color direction matchingassumption may hold true for a scanner that includes “approximately” (hesame resolution in all three color channels, it begins to break down forsensors such as consumer digital cameras or sensors in cell phonecameras. Many of these non-optimal sensors use a Bayer pattern (shownbelow) to detect RGB using a single CMOS sensor.

-   -   Bayer Pattern:        -   R G        -   G B

The red samples are interpolated to create a red color plane, andsimilarly for green and blue. The captured image is also usually JPEGcompressed which loses more information particularly in the bluechannel. In a “non-ideal” case, the best color direction for watermarkdetection is often not precisely matched to the embed color direction.For example, assigning a lower weight to the sub-sampled noisy bluechannel should result in higher detection rates. (When we refer to“weights” (e.g., “4:1:1”) we generally mean that the red channel isweighted 4 times to the green and blue channels.).

Weights can be determined to compensate for lost information due to,e.g., compression and known channel noise.

The optimum detector weights can be estimated by estimating the relativenoise and sampling rates of the color channels or determined empiricallyby measuring the detection rate on an embedded image with a set ofdifferent weights that lend towards selecting more green and red. Inthis regard, a detector is calibrated according to sampled orempirically determined sensor characteristics.

Concluding Remarks

The foregoing are just exemplary implementations of the presentinvention. It will be recognized that there are a great number ofvariations on these basic themes. The foregoing illustrates but a fewapplications of the detailed technology. There arc many others.

To provide a comprehensive disclosure without unduly lengthening thisspecification, applicants incorporate by reference, in their entireties,the disclosures of the above-cited patent documents. The particularcombinations of elements and features in the above-detailed embodimentsare exemplary only; the interchanging and substitution of theseteachings with other teachings in this application and theincorporated-by-reference patents/applications are also contemplated.

The section headings in this patent document are provided for thereader's convenience and should not be interpreted as limiting the scopeof the invention. Subject matter found under one subject heading can hereadily combined with subject matter found under another subjectheading.

There are many embodiments discussed herein which may benefit from theinclusion of two or more watermarks.

The methods, processes, and systems described above may be implementedin hardware, software or a combination of hardware and software. Forexample, the embedding and detection processes may be implemented in aprogrammable computer or a special purpose digital circuit. Similarly,such watermarking method may be implemented in software, firmware,hardware, or combinations of software, firmware and hardware. Themethods and processes described above may be implemented in programsexecuted from a system's memory (a computer readable medium, such as anelectronic, optical or magnetic storage device).

In view of the wide variety of embodiments to which the principles andfeatures discussed above can be applied, it should be apparent that thedetailed embodiments are illustrative only and should not be taken aslimiting the scope of the invention.

What is claimed is:
 1. In a watermark detector, a method of decodingauxiliary information encoded in an image or video comprising: receivingdata representing the image or video, wherein the data comprises firstdata corresponding to a first color channel, second data correspondingto a second color channel and third data corresponding to a third colorchannel; weighting the first data, the second data and the third dataaccording to at least the following two factors: i) a color directionbiased toward an anticipated embedding direction; and ii) anticipatedimage or video distortion introduced to the first data, second data orthird data through scanning or signal processing; and determining fromweighted first data, weighted second data and weighted third data,changes in an image or video attribute, in which the auxiliaryinformation is conveyed through the changes to sample valuesrepresenting the image or video.
 2. The method of claim 1 in which thedata is weighted for detection in a blue-yellow direction.
 3. The methodof claim 1 in which the first data, second data and third datarespectively correspond to red, green and blue.
 4. The method of claim 1in which the anticipated image or video distortion is attributable to atleast one of compression, signal processing or signal sampling.
 5. Anon-transitory computer readable medium comprising instructions storedtherein, said instructions cause an electronic processor to perform themethod of claim
 1. 6. A programmed computing device comprisinginstructions stored in memory to cause said programmed computing deviceto perform the method of claim
 1. 7. An apparatus comprising: electronicmemory for buffering data representing an image or video, in which thedata comprises first data corresponding to a first color channel, seconddata corresponding to a second color channel and third datacorresponding to a third color channel; an electronic processorprogrammed for: weighting the first data, the second data and the thirddata according to at least the following two factors: i) a colordirection biased toward an anticipated embedding direction; and ii)anticipated image or video distortion introduced to the first data,second data or third data through scanning or signal processing; anddetermining from weighted first data, weighted second data and weightedthird data, changes in an image or video attribute, in which theauxiliary information is conveyed through the changes to sample valuesrepresenting the image or video.
 8. The apparatus of claim 7 in whichthe electronic processor is operating to perform at least one of thefunctions recited therein.